Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add flags.deprecate_package_materialization_builtin_override #9956

Merged
merged 10 commits into from
Apr 23, 2024

Conversation

MichelleArk
Copy link
Contributor

@MichelleArk MichelleArk commented Apr 16, 2024

resolves #10007

Problem

Materializations defined in third-party packages are included in the search order for materialization macros. This is difficult to detect, difficult to prevent, and it works differently from other macros.

We should remove third-party packages from the search order.

This change needs to go behind a behavior-change flag:Legacy behaviors | dbt Developer Hub . See #9366 for an implementation of a behavior-change flag.

Solution

Introduce flags.deprecate_package_materialization_builtin_override.

when the flag is False (legacy behaviour):

  • fire a deprecation event and warning and allow the override
    when the flag is True (soon-to-be default behaviour)
  • get the next candidate, only searching for candidates with core and root localities

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
  • This PR includes type annotations for new and modified functions

@cla-bot cla-bot bot added the cla:yes label Apr 16, 2024
Copy link
Contributor

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

Copy link

codecov bot commented Apr 16, 2024

Codecov Report

Attention: Patch coverage is 92.30769% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 88.14%. Comparing base (27943a5) to head (3e38950).
Report is 1 commits behind head on main.

Files Patch % Lines
core/dbt/contracts/graph/manifest.py 90.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9956      +/-   ##
==========================================
- Coverage   88.17%   88.14%   -0.03%     
==========================================
  Files         181      181              
  Lines       22614    22625      +11     
==========================================
+ Hits        19939    19943       +4     
- Misses       2675     2682       +7     
Flag Coverage Δ
integration 85.44% <92.30%> (-0.13%) ⬇️
unit 62.26% <92.30%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@MichelleArk MichelleArk changed the title first pass: flags.allow_materialization_overrides Add flags.deprecate_package_materialization_builtin_override Apr 22, 2024
@@ -355,7 +355,7 @@ def set_common_global_flags(self):
# This is here to prevent mypy from complaining about all of the
# attributes which we added dynamically.
def __getattr__(self, name: str) -> Any:
return super().__get_attribute__(name) # type: ignore
return super().__getattribute__(name) # type: ignore
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This appears to have been broken all along, but was never reached in testing

)
else:
materialization_candidate = candidates.last_candidate(
valid_localities=[Locality.Core, Locality.Root]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

was not able to do this lower-level using the _find_macros_by_name filter parameter because that filter only has a view of the macro candidate being iterated over. For this check, we need to exclude imported materialization macros only if the materialization is builtin in order to continue supporting custom materializations from packages (tested)

@MichelleArk MichelleArk marked this pull request as ready for review April 22, 2024 21:42
@MichelleArk MichelleArk requested a review from a team as a code owner April 22, 2024 21:42
@MichelleArk
Copy link
Contributor Author

MichelleArk commented Apr 22, 2024

Note: we'd like to backport these changes but it will very likely need to be manual since ProjectFlags does not exist in 1.6 or 1.7.

(WIP) Manual backports, starting with backporting changes in #9971
1.6: #10008
1.7: #9998

Copy link
Member

@emmyoop emmyoop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good - just a changelog comment.

.changes/unreleased/Features-20240422-173703.yaml Outdated Show resolved Hide resolved
@MichelleArk
Copy link
Contributor Author

codecov/project is hanging waiting for a response but I can see the results passing on: https://app.codecov.io/gh/dbt-labs/dbt-core/pull/9956
Screenshot 2024-04-23 at 3 54 03 PM

@MichelleArk MichelleArk merged commit 2d33655 into main Apr 23, 2024
61 of 62 checks passed
@MichelleArk MichelleArk deleted the arky/allow_materialization_overrides branch April 23, 2024 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Remove installed (third-party) packages from search order for materialization macros
2 participants